Bug#121063 Members of one group assign different GNOs to the same transaction - #736
Bug#121063 Members of one group assign different GNOs to the same transaction#736matias-sanchez wants to merge 3 commits into
Conversation
oracle-contributor-agreement
Bot
added
the
OCA Verified
github-actions
Bot
added
Replication
jujose-1
left a comment
jujose-1
left a comment
There was a problem hiding this comment.
Hi Matias,
Thank you for the contribution and for working on this fix. I’ve left a few review comments for your consideration. Please take a look when you have a chance.
Regards,
Justin Jose
288df77 to
267fed8
Compare
|
Hi @jujose-1, thanks for taking the time to review this and for the comments. Added the requested changes. |
jujose-1
left a comment
jujose-1
left a comment
There was a problem hiding this comment.
Hi @matias-sanchez ,
Thank you for updating the patch and addressing the earlier feedback. The production change looks good to me. I’ve left one additional comment regarding the unit-test setup.
I’ll also wait for @tiagoportelajorge 's review and any additional feedback he may have.
Thank you again for working on this.
Regards,
Justin Jose
|
My proposal is to add two debug asserts:
|
|
Thanks @kamil-holubicki . However, I'm not sure on how to safely add these asserts, because the issue covered is on multi primary and I'm not sure how to add that assert so that it does not fire when |
|
Thanks, @matias-sanchez , for updating the unit test. The revised test now uses valid memberships and verifies the same reservation across the view change. This looks good to me. Thanks also, @kamil-holubicki, for the suggestions.
My understanding is that the proposed assertions would cover the scenario reported in this bug, but they may not hold for remotely allocated synodes, where the synode can legitimately carry the allocating leader’s node index rather than the proposer’s. Given that the reported issue involves a locally allocated reservation, and the current fix scopes the ownership validation to that path, I suggest keeping this PR focused on that change. The acceptor side does not currently have the allocation provenance needed to apply the same invariant safely, so I would prefer not to add the proposed assertions here. |
…t GNOs to the same transaction https://perconadev.atlassian.net/browse/PS-11530 Upstream bug: Bug#121063: https://bugs.mysql.com/bug.php?id=121063 Problem: Two members of the same group assign different GNOs to the same transaction, in multi-primary mode, when a member leaves and rejoins during a rolling restart under write load. The group splits into internally consistent sets that disagree on the GTID of the same transaction. It can surface on the `group_replication_applier` channel as Error_code 1032 or 1062, or stay silent with every member ONLINE and the disagreement present only in the binary logs. Cause: A reserved synode is only ours while we still hold the node index it was reserved under. `local_synode_allocator` stamps the member's current index into the synode, `synode.node = my_nodeno`. Still inside `reserve_synode_number`, the task yields in the `while (too_far(*msgno))` loop at `TIMED_TASK_WAIT`. During that yield, `site_install_action` reassigns `site->nodeno`. The reservation still carries the old index, so `proposer_task` brands and proposes into a slot that now belongs to another node. The header comment of `xcom_base.cc` states the rule: only node N may propose a value for synode {X N}. With two proposers on the same slot at `cnt=0`, `acceptor.promise` is never raised, since it is assigned in exactly one place, inside `handle_simple_prepare`, which is the phase 1 decision. Both proposals are accepted, both are learned, and `handle_learn` keeps whichever LEARN arrived first because of the `/* Avoid re-learn */` guard. Members that heard different values first deliver different payloads. Solution: Before proposing, verify the reservation still carries the member's own node index. If it does not, drop it through `retry_new` and take a new one. This is the same check `incr_msgno` already makes whenever it advances, with the comment `In case site and node number has changed`. The client transaction is not lost, it goes out in a slot that does belong to the member. Ported from upstream MySQL PR mysql/mysql-server#736 by Matias Sanchez. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
3 participants
Bug#121063: https://bugs.mysql.com/bug.php?id=121063
What happens
Two members of the same group assign different GNOs to the same transaction, in multi-primary mode, when a member leaves and rejoins during a rolling restart under write load. The group splits into internally consistent sets that disagree on the GTID of the same transaction. It can surface on the
group_replication_applierchannel as Error_code 1032 or 1062, or stay silent with every member ONLINE and the disagreement present only in the binary logs.Reproduced on 8.4.8, 8.4.11 and 9.7.2. A self contained reproducer is attached to the bug.
Where it starts
A reserved synode is only ours while we still hold the node index it was reserved under.
local_synode_allocatorstamps the member's current index into the synode,synode.node = my_nodeno. Still insidereserve_synode_number, the task yields in thewhile (too_far(*msgno))loop atTIMED_TASK_WAIT. During that yield,site_install_actionreassignssite->nodeno. The reservation still carries the old index, soproposer_taskbrands and proposes into a slot that now belongs to another node.The header comment of
xcom_base.ccstates the rule at line 107: only node N may propose a value for synode {X N}. With two proposers on the same slot atcnt=0,acceptor.promiseis never raised, since it is assigned in exactly one place, insidehandle_simple_prepare, which is the phase 1 decision. Both proposals are accepted, both are learned, andhandle_learnkeeps whichever LEARN arrived first because of the/* Avoid re-learn */guard. Members that heard different values first deliver different payloads.A bpftrace trace of one such slot followed from ACCEPT through LEARN to DELIVER is in the bug, posted 28 Aug 2026.
The change
Before proposing, verify the reservation still carries the member's own node index. If it does not, drop it through
retry_newand take a new one.This is the same check
incr_msgno(xcom_base.cc:668) already makes whenever it advances, with the commentIn case site and node number has changed. The client transaction is not lost, it goes out in a slot that does belong to the member.Testing
In the lab the anomaly goes to zero: 0 stale proposals out of 15,523,048, against 993 predicted from the unpatched rate, and 0 divergences out of 5,667,504 transactions, against 17.1 expected. No run carrying the patch has reproduced the divergence, on two hosts.
Performance: 12 runs per arm across two hosts, load only, no restarts, no emulated latency, comparing the patched plugin against the same source built without the patch and against the stock image. Ratio of median successful operations per 10 s window, bootstrap over runs: patched over unpatched [0.9998, 1.0004], patched over stock [0.9994, 1.0003]. A positive control with a known throttle was detected at [0.9944, 0.9953], so the method resolves differences of about 0.5%. The workload is rate limited, so this measures behaviour under a production-like load and not peak capacity.
I do not have an MTR test for this. The reproduction needs seven members, rolling restarts and sustained write load, which does not fit the standard framework. The reproducer attached to the bug builds that environment and decides the verdict by decoding every member's binary log and comparing all 21 pairs.